Measuring the Performance of Automated Planning Systems

To appear in Performance Metrics for Intelligent Systems Workshop (PerMIS ’04). In this paper, we describe existing performance measures for automated planning algorithms, and discuss the limitations and biases inherent in those performance measures. We point out the importance of developing a performance measure that explicitly the restrictive assumptions on which a planning algorithm depends, and we propose a composite performance measure based on three factors: • the scope of the planning algorithm: which set of restrictive assumption are needed and which can be lifted, • the control knowledge and tuning required for each planning do-